Information Theory and Machine Learning

Machine learning techniques are becoming increasingly useful primarily with the rapid development of Internet. A variety of machine learning methods have drawn inspirations or borrowed ideas from information theory. In this paper, we present a survey of such interactions between machine learning and information theory. Four important areas of machine learning are examined from the perspective of information theory: clustering, semi-supervised learning, feature selection and metric learning. Finally, we conclude this paper by summarizing the way how information theory interacts with machine learning and pointing out some open questions.